Earth Engine Datalab Initialization

This notebook demonstrates how to configure a new Google Cloud Datalab instance for use with Earth Engine. You should only need to do this once when you first configure your instance.

First we install a variety of required and useful libraries, including the Earth Engine SDK itself. This may take a couple of mintues, and will generate lots of output which you can usually ignore.


In [1]:
%%bash
apt-get update
apt-get install -y libffi-dev libssl-dev python-dev python-pip python-imaging python-gdal
pip install cryptography google-api-python-client earthengine-api


Requirement already satisfied (use --upgrade to upgrade): cryptography in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): google-api-python-client in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): earthengine-api in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): setuptools>=1.0 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.8 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): uritemplate<1,>=0.6 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from google-api-python-client)
Requirement already satisfied (use --upgrade to upgrade): httplib2<1,>=0.8 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from google-api-python-client)
Requirement already satisfied (use --upgrade to upgrade): oauth2client in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from google-api-python-client)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL>=0.11 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from earthengine-api)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography)
Requirement already satisfied (use --upgrade to upgrade): simplejson>=2.5.0 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from uritemplate<1,>=0.6->google-api-python-client)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules>=0.0.5 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from oauth2client->google-api-python-client)
Requirement already satisfied (use --upgrade to upgrade): rsa>=3.1.4 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from oauth2client->google-api-python-client)
bash: line 1: apt-get: command not found
bash: line 2: apt-get: command not found

2. Authorize access to the Earth Engine API

This walks through the same flow that is used by the "earthengine authenticate" command, but it works even when you do not have easy access to an interactive shell on the remote host. These should become available as helper functions in the Earth Engine client library, but for now you can still do it by hand.

Start by importing the Python libraries that we're going to use to configure the Earth Engine credentials.


In [2]:
import ee
import errno
import IPython
import json
import os
import urllib
import urllib2

These helper functions implement the authentication flow.


In [3]:
def get_authorization_url():
  # This redirect_uri prompts you to copy and paste a code after successful authorization.
  return 'https://accounts.google.com/o/oauth2/auth?' + urllib.urlencode({
      'client_id': ee.oauthinfo.OAuthInfo.CLIENT_ID,
      'scope': ee.oauthinfo.OAuthInfo.SCOPE,
      'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob',
      'response_type': 'code',
  })
  
def get_refresh_token(auth_code):
  args = {
    'code': auth_code,
    'client_id': ee.oauthinfo.OAuthInfo.CLIENT_ID,
    'client_secret': ee.oauthinfo.OAuthInfo.CLIENT_SECRET,
    'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob',
    'grant_type': 'authorization_code',
  }
  response = urllib2.urlopen('https://accounts.google.com/o/oauth2/token',
                             urllib.urlencode(args)).read()
  return json.loads(response)['refresh_token']

def save_token(refresh_token):
  credentials_path = ee.oauthinfo.OAuthInfo.credentials_path()
  try:
    os.makedirs(os.path.dirname(credentials_path))
  except OSError, e:
    if e.errno != errno.EEXIST:
      raise
  json.dump({'refresh_token': refresh_token}, open(credentials_path, 'w'))
  print 'Successfully saved authorization to %s' % credentials_path

Running this next command will print a URL that you can click on to begin the authentication flow. Copy the code that it generates and come back here for the next step.


In [4]:
print(get_authorization_url())


https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com

Finally, paste the auth code that you were given into this next command (in place of "YOUR_AUTH_TOKEN_GOES_HERE") and run it to save your credentials on this Datalab instance.


In [6]:
save_token(get_refresh_token('https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com'))


---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-6-668e7346224c> in <module>()
----> 1 save_token(get_refresh_token('https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com'))

<ipython-input-3-80893d2d5690> in get_refresh_token(auth_code)
     17   }
     18   response = urllib2.urlopen('https://accounts.google.com/o/oauth2/token',
---> 19                              urllib.urlencode(args)).read()
     20   return json.loads(response)['refresh_token']
     21 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in urlopen(url, data, timeout)
    124     if _opener is None:
    125         _opener = build_opener()
--> 126     return _opener.open(url, data, timeout)
    127 
    128 def install_opener(opener):

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in open(self, fullurl, data, timeout)
    404         for processor in self.process_response.get(protocol, []):
    405             meth = getattr(processor, meth_name)
--> 406             response = meth(req, response)
    407 
    408         return response

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in http_response(self, request, response)
    517         if not (200 <= code < 300):
    518             response = self.parent.error(
--> 519                 'http', request, response, code, msg, hdrs)
    520 
    521         return response

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in error(self, proto, *args)
    442         if http_err:
    443             args = (dict, 'default', 'http_error_default') + orig_args
--> 444             return self._call_chain(*args)
    445 
    446 # XXX probably also want an abstract factory that knows when it makes

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in _call_chain(self, chain, kind, meth_name, *args)
    376             func = getattr(handler, meth_name)
    377 
--> 378             result = func(*args)
    379             if result is not None:
    380                 return result

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.pyc in http_error_default(self, req, fp, code, msg, hdrs)
    525 class HTTPDefaultErrorHandler(BaseHandler):
    526     def http_error_default(self, req, fp, code, msg, hdrs):
--> 527         raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    528 
    529 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 400: Bad Request

3. Take it for a spin

Finally, test out our new credentials by doing something with Earth Engine. This should display a simple thumbnail of global SRTM terrain data.


In [7]:
ee.Initialize()
srtm = ee.Image('USGS/SRTMGL1_003')
srtm_vis = srtm.visualize(min=0, max=5000, gamma=1.6)
srtm_thumb = ee.data.getThumbnail({'image':srtm_vis.serialize()})
IPython.display.display(IPython.display.Image(srtm_thumb))


---------------------------------------------------------------------------
HttpAccessTokenRefreshError               Traceback (most recent call last)
<ipython-input-7-74e02f8d1b64> in <module>()
----> 1 ee.Initialize()
      2 srtm = ee.Image('USGS/SRTMGL1_003')
      3 srtm_vis = srtm.visualize(min=0, max=5000, gamma=1.6)
      4 srtm_thumb = ee.data.getThumbnail({'image':srtm_vis.serialize()})
      5 IPython.display.display(IPython.display.Image(srtm_thumb))

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ee/__init__.pyc in Initialize(credentials, opt_url)
     82   data.initialize(credentials, (opt_url + '/api' if opt_url else None), opt_url)
     83   # Initialize the dynamically loaded functions on the objects that want them.
---> 84   ApiFunction.initialize()
     85   Element.initialize()
     86   Image.initialize()

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ee/apifunction.pyc in initialize(cls)
    143     """Initializes the list of signatures from the Earth Engine front-end."""
    144     if not cls._api:
--> 145       signatures = data.getAlgorithms()
    146       api = {}
    147       for name, sig in signatures.iteritems():

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ee/data.pyc in getAlgorithms()
    339                 is not specified.
    340   """
--> 341   return send_('/algorithms', {}, 'GET')
    342 
    343 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ee/data.pyc in send_(path, params, opt_method, opt_raw)
    597   try:
    598     response, content = http.request(url, method=opt_method, body=payload,
--> 599                                      headers=headers)
    600   except httplib2.HttpLib2Error, e:
    601     raise ee_exception.EEException(

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    596                 logger.info('Attempting refresh to obtain '
    597                             'initial access_token')
--> 598                 self._refresh(request_orig)
    599 
    600             # Clone and modify the request headers to add the appropriate

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client/client.pyc in _refresh(self, http_request)
    862         """
    863         if not self.store:
--> 864             self._do_refresh_request(http_request)
    865         else:
    866             self.store.acquire_lock()

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    931             except (TypeError, ValueError):
    932                 pass
--> 933             raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
    934 
    935     def _revoke(self, http_request):

HttpAccessTokenRefreshError: invalid_grant

In [8]:
!ls


20141103-World-Origin.mov
26fb4HJKeWtAAAAAElFTkSuQmCC.png
A1--THINGS TO DO--9 June 2016.docx
Behrenfeld_Falkowski_1997 (1).pdf
Chlorophyll_2015_01.tiff
Earth Engine Datalab Initialization.ipynb
Earth Engine and TensorFlow.ipynb
Global Density Raster High Res.py
Hobet_timelapse.mp4
Jobs
MY1DMM_CHLORA_2015-01-01_rgb_3600x1800.FLOAT.TIFF
Rast_high_res.py
Screen Shot 2016-06-13 at 10.00.58 PM.png
Screen Shot 2016-06-13 at 10.03.07 PM.png
Screen Shot 2016-06-13 at 10.11.10 PM.png
Screen Shot 2016-06-13 at 10.19.11 PM.png
Screen Shot 2016-06-13 at 10.47.14 PM.png
Screen Shot 2016-06-13 at 10.48.02 PM.png
Screen Shot 2016-06-13 at 10.51.39 PM.png
Screen Shot 2016-06-13 at 10.51.48 PM.png
Screen Shot 2016-06-13 at 10.52.08 PM.png
Screen Shot 2016-06-13 at 2.25.54 PM (2).png
Screen Shot 2016-06-13 at 2.25.54 PM.png
Screen Shot 2016-06-13 at 2.25.57 PM (2).png
Screen Shot 2016-06-13 at 2.25.57 PM.png
Screen Shot 2016-06-13 at 9.48.57 PM.png
Screen Shot 2016-06-14 at 1.37.06 PM.png
Screen Shot 2016-06-14 at 1.37.23 PM.png
Screen Shot 2016-06-14 at 10.44.00 AM.png
Screen Shot 2016-06-14 at 10.49.52 AM.png
Screen Shot 2016-06-14 at 10.50.01 AM.png
Screen Shot 2016-06-14 at 11.02.32 AM.png
Screen Shot 2016-06-14 at 11.05.59 AM.png
Screen Shot 2016-06-14 at 11.07.43 AM.png
Screen Shot 2016-06-14 at 11.38.34 AM.png
Screen Shot 2016-06-14 at 11.39.12 AM.png
Screen Shot 2016-06-14 at 11.39.28 AM.png
Screen Shot 2016-06-14 at 12.41.07 PM.png
Screen Shot 2016-06-14 at 7.43.58 AM.png
Screen Shot 2016-06-15 at 5.35.53 PM.png
big_query.py
ee_test.py
global_raster.tiff
greece
imgres.jpg
oc_partner_logo.png
to organize
unnamed.png
~$125179.kmz
~$441220316.xlsx
~$dams2.xlsx
~$merge.xlsx
~$population.xlsx
~$test.xlsx

In [ ]: